In order to study the effects of frustration on the magnetic properties of the antiferromagnetic triangular lattice LiNi1-xCoxO2, we measured the temperature dependence of both the magnetization and the AC Susceptibility. With a random substitution of non-magnetic Co ions for Ni ions, the characteristic temperatures T(N1) (congruent-to 240 K) and T(N2)(congruent-to 35 K) gradually shifted to higher and lower temperatures, respectively, and the magnetic moment per Ni ion was drastically enhanced between T(N1) and T(N2). The spin angular momentum of the Ni3+ ion, however, remains at about 0.5, confirming the low-spin state over the whole range of concentrations. We also performed neutron-scattering experiments and found that the intensity of diffuse scattering in the forward directions at low temperatures decreases on substitution. A new model for the magnetic interactions in LiNiO2 is proposed on the basis of these results.